-
Open
-
Open
-
Open
-
Open
-
Open
-
Open
A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition. (arXiv:2105.04187v4 [cs.IT] UPDATED)
Selecting a minimal feature set that is maximally informative about a target
variable is a central task in machine learning and statistics. Information
theory provides a powerful framework for formulating feature selection
algorithms -- yet, a rigorous, information-theoretic definition of feature
relevancy, which accounts for feature interactions such as redundant and
synergistic contributions, is still missing. We argue that this lack is
inherent to classical information theory which does not provide measures to
decompose the information a set of variables provides about a target into
unique, redundant, and synergistic contributions. Such a decomposition has been
introduced only recently by the partial information decomposition (PID)
framework. Using PID, we clarify why feature selection is a conceptually
difficult problem when approached using information theory and provide a novel
definition of feature relevancy and redundancy in PID terms. From this
definition, we show that the conditional mutual information (CMI) maximizes
relevancy while minimizing redundancy and propose an iterative, CMI-based
algorithm for practical feature selection. We demonstrate the power of our
CMI-based algorithm in comparison to the unconditional mutual information on
benchmark examples and provide corresponding PID estimates to highlight how PID
allows to quantify information contribution of features and their interactions
in feature-selection problems.
( 3
min )
Explainable Reinforcement Learning via a Causal World Model. (arXiv:2305.02749v1 [cs.LG])
Generating explanations for reinforcement learning (RL) is challenging as
actions may produce long-term effects on the future. In this paper, we develop
a novel framework for explainable RL by learning a causal world model without
prior knowledge of the causal structure of the environment. The model captures
the influence of actions, allowing us to interpret the long-term effects of
actions through causal chains, which present how actions influence
environmental variables and finally lead to rewards. Different from most
explanatory models which suffer from low accuracy, our model remains accurate
while improving explainability, making it applicable in model-based learning.
As a result, we demonstrate that our causal model can serve as the bridge
between explainability and learning.
( 2
min )
Recent Advances in the Foundations and Applications of Unbiased Learning to Rank. (arXiv:2305.02914v1 [cs.IR])
Since its inception, the field of unbiased learning to rank (ULTR) has
remained very active and has seen several impactful advancements in recent
years. This tutorial provides both an introduction to the core concepts of the
field and an overview of recent advancements in its foundations along with
several applications of its methods. The tutorial is divided into four parts:
Firstly, we give an overview of the different forms of bias that can be
addressed with ULTR methods. Secondly, we present a comprehensive discussion of
the latest estimation techniques in the ULTR field. Thirdly, we survey
published results of ULTR in real-world applications. Fourthly, we discuss the
connection between ULTR and fairness in ranking. We end by briefly reflecting
on the future of ULTR research and its applications. This tutorial is intended
to benefit both researchers and industry practitioners who are interested in
developing new ULTR solutions or utilizing them in real-world applications.
( 2
min )
A Novel Plagiarism Detection Approach Combining BERT-based Word Embedding, Attention-based LSTMs and an Improved Differential Evolution Algorithm. (arXiv:2305.02374v1 [cs.CL])
Detecting plagiarism involves finding similar items in two different sources.
In this article, we propose a novel method for detecting plagiarism that is
based on attention mechanism-based long short-term memory (LSTM) and
bidirectional encoder representations from transformers (BERT) word embedding,
enhanced with optimized differential evolution (DE) method for pre-training and
a focal loss function for training. BERT could be included in a downstream task
and fine-tuned as a task-specific BERT can be included in a downstream task and
fine-tuned as a task-specific structure, while the trained BERT model is
capable of detecting various linguistic characteristics. Unbalanced
classification is one of the primary issues with plagiarism detection. We
suggest a focal loss-based training technique that carefully learns minority
class instances to solve this. Another issue that we tackle is the training
phase itself, which typically employs gradient-based methods like
back-propagation for the learning process and thus suffers from some drawbacks,
including sensitivity to initialization. To initiate the BP process, we suggest
a novel DE algorithm that makes use of a clustering-based mutation operator.
Here, a winning cluster is identified for the current DE population, and a
fresh updating method is used to produce potential answers. We evaluate our
proposed approach on three benchmark datasets ( MSRP, SNLI, and SemEval2014)
and demonstrate that it performs well when compared to both conventional and
population-based methods.
( 3
min )
Learning to Detect Novel and Fine-Grained Acoustic Sequences Using Pretrained Audio Representations. (arXiv:2305.02382v1 [cs.SD])
This work investigates pretrained audio representations for few shot Sound
Event Detection. We specifically address the task of few shot detection of
novel acoustic sequences, or sound events with semantically meaningful temporal
structure, without assuming access to non-target audio. We develop procedures
for pretraining suitable representations, and methods which transfer them to
our few shot learning scenario. Our experiments evaluate the general purpose
utility of our pretrained representations on AudioSet, and the utility of
proposed few shot methods via tasks constructed from real-world acoustic
sequences. Our pretrained embeddings are suitable to the proposed task, and
enable multiple aspects of our few shot framework.
( 2
min )
RCP-RF: A Comprehensive Road-car-pedestrian Risk Management Framework based on Driving Risk Potential Field. (arXiv:2305.02493v1 [cs.LG])
Recent years have witnessed the proliferation of traffic accidents, which led
wide researches on Automated Vehicle (AV) technologies to reduce vehicle
accidents, especially on risk assessment framework of AV technologies. However,
existing time-based frameworks can not handle complex traffic scenarios and
ignore the motion tendency influence of each moving objects on the risk
distribution, leading to performance degradation. To address this problem, we
novelly propose a comprehensive driving risk management framework named RCP-RF
based on potential field theory under Connected and Automated Vehicles (CAV)
environment, where the pedestrian risk metric are combined into a unified
road-vehicle driving risk management framework. Different from existing
algorithms, the motion tendency between ego and obstacle cars and the
pedestrian factor are legitimately considered in the proposed framework, which
can improve the performance of the driving risk model. Moreover, it requires
only O(N 2) of time complexity in the proposed method. Empirical studies
validate the superiority of our proposed framework against state-of-the-art
methods on real-world dataset NGSIM and real AV platform.
( 2
min )
Nearly-Linear Time and Streaming Algorithms for Outlier-Robust PCA. (arXiv:2305.02544v1 [cs.LG])
We study principal component analysis (PCA), where given a dataset in
$\mathbb{R}^d$ from a distribution, the task is to find a unit vector $v$ that
approximately maximizes the variance of the distribution after being projected
along $v$. Despite being a classical task, standard estimators fail drastically
if the data contains even a small fraction of outliers, motivating the problem
of robust PCA. Recent work has developed computationally-efficient algorithms
for robust PCA that either take super-linear time or have sub-optimal error
guarantees. Our main contribution is to develop a nearly-linear time algorithm
for robust PCA with near-optimal error guarantees. We also develop a
single-pass streaming algorithm for robust PCA with memory usage nearly-linear
in the dimension.
( 2
min )
Machine Learning Benchmarks for the Classification of Equivalent Circuit Models from Electrochemical Impedance Spectra. (arXiv:2302.03362v2 [cs.LG] UPDATED)
Analysis of Electrochemical Impedance Spectroscopy (EIS) data for
electrochemical systems often consists of defining an Equivalent Circuit Model
(ECM) using expert knowledge and then optimizing the model parameters to
deconvolute various resistance, capacitive, inductive, or diffusion responses.
For small data sets, this procedure can be conducted manually; however, it is
not feasible to manually define a proper ECM for extensive data sets with a
wide range of EIS responses. Automatic identification of an ECM would
substantially accelerate the analysis of large sets of EIS data. We showcase
machine learning methods to classify the ECMs of 9,300 impedance spectra
provided by QuantumScape for the BatteryDEV hackathon. The best-performing
approach is a gradient-boosted tree model utilizing a library to automatically
generate features, followed by a random forest model using the raw spectral
data. A convolutional neural network using boolean images of Nyquist
representations is presented as an alternative, although it achieves a lower
accuracy. We publish the data and open source the associated code. The
approaches described in this article can serve as benchmarks for further
studies. A key remaining challenge is the identifiability of the labels,
underlined by the model performances and the comparison of misclassified
spectra.
( 3
min )
Integrating Psychometrics and Computing Perspectives on Bias and Fairness in Affective Computing: A Case Study of Automated Video Interviews. (arXiv:2305.02629v1 [cs.LG])
We provide a psychometric-grounded exposition of bias and fairness as applied
to a typical machine learning pipeline for affective computing. We expand on an
interpersonal communication framework to elucidate how to identify sources of
bias that may arise in the process of inferring human emotions and other
psychological constructs from observed behavior. Various methods and metrics
for measuring fairness and bias are discussed along with pertinent implications
within the United States legal context. We illustrate how to measure some types
of bias and fairness in a case study involving automatic personality and
hireability inference from multimodal data collected in video interviews for
mock job applications. We encourage affective computing researchers and
practitioners to encapsulate bias and fairness in their research processes and
products and to consider their role, agency, and responsibility in promoting
equitable and just systems.
( 2
min )
Using Language Models on Low-end Hardware. (arXiv:2305.02350v1 [cs.CL])
This paper evaluates the viability of using fixed language models for
training text classification networks on low-end hardware. We combine language
models with a CNN architecture and put together a comprehensive benchmark with
8 datasets covering single-label and multi-label classification of topic,
sentiment, and genre. Our observations are distilled into a list of trade-offs,
concluding that there are scenarios, where not fine-tuning a language model
yields competitive effectiveness at faster training, requiring only a quarter
of the memory compared to fine-tuning.
( 2
min )
A Novel Evolutionary Algorithm for Hierarchical Neural Architecture Search. (arXiv:2107.08484v2 [cs.NE] UPDATED)
In this work, we propose a novel evolutionary algorithm for neural
architecture search, applicable to global search spaces. The algorithm's
architectural representation organizes the topology in multiple hierarchical
modules, while the design process exploits this representation, in order to
explore the search space. We also employ a curation system, which promotes the
utilization of well performing sub-structures to subsequent generations. We
apply our method to Fashion-MNIST and NAS-Bench101, achieving accuracies of
$93.2\%$ and $94.8\%$ respectively in a relatively small number of generations.
( 2
min )
ExeKGLib: Knowledge Graphs-Empowered Machine Learning Analytics. (arXiv:2305.02966v1 [cs.LG])
Many machine learning (ML) libraries are accessible online for ML
practitioners. Typical ML pipelines are complex and consist of a series of
steps, each of them invoking several ML libraries. In this demo paper, we
present ExeKGLib, a Python library that allows users with coding skills and
minimal ML knowledge to build ML pipelines. ExeKGLib relies on knowledge graphs
to improve the transparency and reusability of the built ML workflows, and to
ensure that they are executable. We demonstrate the usage of ExeKGLib and
compare it with conventional ML code to show its benefits.
( 2
min )
-
Open
Impact Study of Numerical Discretization Accuracy on Parameter Reconstructions and Model Parameter Distributions. (arXiv:2305.02663v1 [physics.comp-ph])
Numerical models are used widely for parameter reconstructions in the field
of optical nano metrology. To obtain geometrical parameters of a nano
structured line grating, we fit a finite element numerical model to an
experimental data set by using the Bayesian target vector optimization method.
Gaussian process surrogate models are trained during the reconstruction.
Afterwards, we employ a Markov chain Monte Carlo sampler on the surrogate
models to determine the full model parameter distribution for the reconstructed
model parameters. The choice of numerical discretization parameters, like the
polynomial order of the finite element ansatz functions, impacts the numerical
discretization error of the forward model. In this study we investigate the
impact of numerical discretization parameters of the forward problem on the
reconstructed parameters as well as on the model parameter distributions. We
show that such a convergence study allows to determine numerical parameters
which allow for efficient and accurate reconstruction results.
( 2
min )
Using interpretable boosting algorithms for modeling environmental and agricultural data. (arXiv:2305.02699v1 [stat.ML])
We describe how interpretable boosting algorithms based on ridge-regularized
generalized linear models can be used to analyze high-dimensional environmental
data. We illustrate this by using environmental, social, human and biophysical
data to predict the financial vulnerability of farmers in Chile and Tunisia
against climate hazards. We show how group structures can be considered and how
interactions can be found in high-dimensional datasets using a novel 2-step
boosting approach. The advantages and efficacy of the proposed method are shown
and discussed. Results indicate that the presence of interaction effects only
improves predictive power when included in two-step boosting. The most
important variable in predicting all types of vulnerabilities are natural
assets. Other important variables are the type of irrigation, economic assets
and the presence of crop damage of near farms.
( 2
min )
Semisupervised regression in latent structure networks on unknown manifolds. (arXiv:2305.02473v1 [stat.ML])
Random graphs are increasingly becoming objects of interest for modeling
networks in a wide range of applications. Latent position random graph models
posit that each node is associated with a latent position vector, and that
these vectors follow some geometric structure in the latent space. In this
paper, we consider random dot product graphs, in which an edge is formed
between two nodes with probability given by the inner product of their
respective latent positions. We assume that the latent position vectors lie on
an unknown one-dimensional curve and are coupled with a response covariate via
a regression model. Using the geometry of the underlying latent position
vectors, we propose a manifold learning and graph embedding technique to
predict the response variable on out-of-sample nodes, and we establish
convergence guarantees for these responses. Our theoretical results are
supported by simulations and an application to Drosophila brain data.
( 2
min )
A Cross Validation Framework for Signal Denoising with Applications to Trend Filtering, Dyadic CART and Beyond. (arXiv:2201.02654v3 [math.ST] UPDATED)
This paper formulates a general cross validation framework for signal
denoising. The general framework is then applied to nonparametric regression
methods such as Trend Filtering and Dyadic CART. The resulting cross validated
versions are then shown to attain nearly the same rates of convergence as are
known for the optimally tuned analogues. There did not exist any previous
theoretical analyses of cross validated versions of Trend Filtering or Dyadic
CART. To illustrate the generality of the framework we also propose and study
cross validated versions of two fundamental estimators; lasso for high
dimensional linear regression and singular value thresholding for matrix
estimation. Our general framework is inspired by the ideas in Chatterjee and
Jafarov (2015) and is potentially applicable to a wide range of estimation
methods which use tuning parameters.
( 2
min )
Mathematical analysis of singularities in the diffusion model under the submanifold assumption. (arXiv:2301.07882v3 [cs.LG] UPDATED)
This paper provide several mathematical analyses of the diffusion model in
machine learning. The drift term of the backwards sampling process is
represented as a conditional expectation involving the data distribution and
the forward diffusion. The training process aims to find such a drift function
by minimizing the mean-squared residue related to the conditional expectation.
Using small-time approximations of the Green's function of the forward
diffusion, we show that the analytical mean drift function in DDPM and the
score function in SGM asymptotically blow up in the final stages of the
sampling process for singular data distributions such as those concentrated on
lower-dimensional manifolds, and is therefore difficult to approximate by a
network. To overcome this difficulty, we derive a new target function and
associated loss, which remains bounded even for singular data distributions. We
illustrate the theoretical findings with several numerical examples.
( 2
min )
Nearly-Linear Time and Streaming Algorithms for Outlier-Robust PCA. (arXiv:2305.02544v1 [cs.LG])
We study principal component analysis (PCA), where given a dataset in
$\mathbb{R}^d$ from a distribution, the task is to find a unit vector $v$ that
approximately maximizes the variance of the distribution after being projected
along $v$. Despite being a classical task, standard estimators fail drastically
if the data contains even a small fraction of outliers, motivating the problem
of robust PCA. Recent work has developed computationally-efficient algorithms
for robust PCA that either take super-linear time or have sub-optimal error
guarantees. Our main contribution is to develop a nearly-linear time algorithm
for robust PCA with near-optimal error guarantees. We also develop a
single-pass streaming algorithm for robust PCA with memory usage nearly-linear
in the dimension.
( 2
min )
FastAMI -- a Monte Carlo Approach to the Adjustment for Chance in Clustering Comparison Metrics. (arXiv:2305.03022v1 [cs.LG])
Clustering is at the very core of machine learning, and its applications
proliferate with the increasing availability of data. However, as datasets
grow, comparing clusterings with an adjustment for chance becomes
computationally difficult, preventing unbiased ground-truth comparisons and
solution selection. We propose FastAMI, a Monte Carlo-based method to
efficiently approximate the Adjusted Mutual Information (AMI) and extend it to
the Standardized Mutual Information (SMI). The approach is compared with the
exact calculation and a recently developed variant of the AMI based on pairwise
permutations, using both synthetic and real data. In contrast to the exact
calculation our method is fast enough to enable these adjusted
information-theoretic comparisons for large datasets while maintaining
considerably more accurate results than the pairwise approach.
( 2
min )
Discovering Communication Pattern Shifts in Large-Scale Networks using Encoder Embedding and Vertex Dynamics. (arXiv:2305.02381v1 [cs.SI])
The analysis of large-scale time-series network data, such as social media
and email communications, remains a significant challenge for graph analysis
methodology. In particular, the scalability of graph analysis is a critical
issue hindering further progress in large-scale downstream inference. In this
paper, we introduce a novel approach called "temporal encoder embedding" that
can efficiently embed large amounts of graph data with linear complexity. We
apply this method to an anonymized time-series communication network from a
large organization spanning 2019-2020, consisting of over 100 thousand vertices
and 80 million edges. Our method embeds the data within 10 seconds on a
standard computer and enables the detection of communication pattern shifts for
individual vertices, vertex communities, and the overall graph structure.
Through supporting theory and synthesis studies, we demonstrate the theoretical
soundness of our approach under random graph models and its numerical
effectiveness through simulation studies.
( 2
min )
